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METHODS AND APPARATUS FOR DECODING IMAGES USING DEDICATED 
HARDWARE CIRCUITRY AND A PROGRAMMABLE PROCESSOR 

FIELD OF THE INVENTION 

The present invention relates to decoding of 
images and, more specifically, to methods and apparatus 
5 for performing video decoding in a system including a 
programmable processor, e.g., a central processing unit 
(CPU) . 

BACKGROUND OF THE INVENTION 

10 

The digital representation of images is 
becoming ever more common. Digital video, e.g., digital 
high definition television (HDTV) , represents one 
application where digital video signals comprise, e.g., a 
15 sequence of digital images. With the increasing use of 

the Internet, the generation, storage and transmission of 
digital video images is likely to continue to increase. 

In order to reduce the amount of data required 
2 0 to represent digital images, such images are often 

encoded, e.g., compressed. Coding of images in the form 
of frames may be performed on an intra- frame basis so 
that the generated encoded image data does not depend on 
data from other images. Intra-frame coding is often 
25 performed using discrete cosine transform (DCT) coding 
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techniques. Data generated from intra-frame coding is 
often called intra-coded data. 



When a sequence of images is being coded, 
5 coding of images may be performed on an inter- frame, as 
well as an intra-frame, basis. Inter-frame coding 
generally involves using motion compensated prediction 
techniques to produce motion vectors which include 
information on how a portion of a subsequent image may be 
O 10 generated using a portion of a preceding or subsequent 
S image as reference data. Motion vector information, once 

J" generated, may be coded, e.g., using differential coding 

y techniques, to reduce the amount of data required to 

~?l represent the motion vector information. 

b 15 

y, When there is little motion between frames, 

motion vectors provide an efficient method of 

%j3 representing portions of image. However, intra-frame 

coding can be more efficient when there is a substantial 
2 0 change from one frame to another. In an attempt to 
maximize coding efficiency, often some portions of an 
inter-coded frame are encoded using intra-frame coding 
techniques while other portions of the same image are 
coded using inter-frame coding, e.g., motion vectors. 
2 5 Those portions of inter-coded frames that use inter-frame 
coding, may also include coded information regarding the 
residual, or correction, image. During decoding of such 
portions of frames, the coded residual data is decoded. 
The decoded residual image data is combined with image 



data generated by performing motion compensated 
predictions using reference frame data and motion 
vectors. Those portions of inter-coded frames that use 
intra- frame coding are decoded without the use of 
5 reference frame data or motion vectors. In this manner, 
a complete inter- coded frame can be generated from the 
intra- and inter-coded data used to represent the frame. 

MPEG-2 is a well known video standard which 
10 includes support for the use of motion compensated 

prediction techniques in addition to transform coding, 
e.g., the use of DCT transforms. The international video 
standard MPEG-2 is described in the International 
Standards Organization document (ISO/IEC 13818-2). MPEG- 
15 2 has been used as the basis for several commercial 
applications including digital video disks (DVD) and 
digital broadcast television. 

In MPEG-2 motion vector information is 
20 differentially encoded for transmission purposes. In 

addition, quantization operations and run length coding 
operations are performed on video data to further reduce 
the amount of data required to represent the images being 
encoded. A scan conversion operation is also normally 
25 performed as part of the MPEG-2 coding process in order 

to convert a set of two dimensional coefficient data into 
a one dimensional data sequence which can be processed, 
stored and/or transmitted. Variable length coding is 
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applied to many of the data elements in order to further 
reduce the number of bits needed to represent an image. 

In order to view images represented using 
5 encoded video data, the image data has to be decoded 
prior to display. MPEG-2 decoding generally involves 
performing operations which are the inverse to those used 
to originally encoded the image data, e.g./ decoding 
usually involves variable length decoding, inverse scan 
10 conversion, inverse quantization, inverse discrete cosine 
transform (IDCT) , motion vector reconstruction and motion 
compensated prediction operations. 

Given the large amount of data used to 
15 represent images, a considerable amount of processing is 
normally required to decode an encoded image. In the 
case of motion vectors, image decoding is complicated by 
the need to access reference frame data, e.g., previously 
decoded frames, in order to generate a current decoded 
2 0 frame. The need to access reference frame data to 
perform motion compensated predictions using motion 
vector information results in the motion compensated 
prediction operations being highly memory intensive. The 
large number of memory access operations associated with 
25 performing motion compensated predictions can often have 
a significant impact on the amount of resources required 
to decode an image . 
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Users of computers are beginning to expect that 
they will be able to decode and display video images, 
e.g., encoded motion pictures, in real time. In order to 
achieve the real time display of encoded video, the 
5 encoded video images, e.g., frames, on average, need to 
be decoded in the same or a smaller amount of time than 
is used to display the images. As discussed above, the 
amount of processing required to decode an image can be 
considerable. Placing real time restraints on decoder 
10 circuitry further complicates matters due to the time 

constraints in which the decoding must be accomplished. 



Generally, known video decoders belong to one 
of three types: 1) those that use dedicated special 

15 purpose integrated decoder circuits coupled with video 

memory to fully decode encoded video data; 2) those that 
use software and a general purpose programmable 
processor, e.g., CPU such as a Pentium processor, to 
fully decode encoded video data; and 3) those that 

2 0 partition the video decoding between a general purpose 
programmable processing unit such as a Pentium processor 
and a graphics processor chip which is used to manage 
and/or perform memory intensive operations such as motion 
compensated prediction. A high speed video or graphics 

2 5 memory is often used in conjunction with the graphic's 
processor to further accelerate image decoding 
operations . 
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In many high definition television sets and 
other video display devices, the first type of known 
decoders are used. That is, dedicated hardware decoder 
circuits are used to decode received encoded video data. 
5 Fig. 1 illustrates a known decoder circuit 100 capable of 
decoding MPEG-2 video signals. 

As illustrated, the video decoder 100 comprises 
a variable length decoder (VLD) circuit 102, an inverse 

10 scan circuit 104, an inverse quantization circuit 106, an 
inverse discrete cosine transform (IDCT) circuit 108, a 
motion compensated prediction circuit 110, and a frame 
store memory 112. Motion vector reconstruction 
circuitry, which is used to reconstruct motion vectors 

15 from encoded motion vector information, is present but 
not explicitly shown in the Fig. 1 illustration. 

In the known system, the VLD circuit 102 
receives MPEG-2 encoded video data and performs a 

2 0 variable length decoding operation on variable length 

encoded data included therein. The inverse scan circuit 
104 is responsible for re-sequencing elements of the 
video data output by the VLD circuit 102 to reverse the 
effect of the scan conversion operation originally used 

25 to convert the two dimensional coefficient data into a 
one dimensional ordering. The inverse quantization 
(I QUANT) circuit 106 performs an inverse quantization 
operation on the quantized values included in the video 
data output by the inverse scan circuit 104. The I QUANT 
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circuit 106 generates a video data stream including DCT 
coefficients. The DCT coefficients are processed by the 
IDCT circuit 108 to generate decoded image date from 
received intra-coded image data. The output of the IDCT 
5 circuit, in the case of intra-coded data, is decoded 

video, e.g., pixel values which are used to control the 
light output of individual pixels. 

As discussed above inter-coded images may 
10 include both intra and inter coded image portions. The 
motion compensated prediction circuit is responsible for 
using previously decoded image data, which has been 
stored in the frame store memory 102, as reference frame 
data. The motion compensated prediction circuit 110 
15 outputs received decoded image data corresponding to 
intra-coded frames as well as decoded image data 
corresponding to inter-coded frames which it generates 
from received motion vectors and/or by combining decoded 
inter-coded image data with decoded image data. 

20 

In the case of personal computers, the computer 
already includes a general purpose processor which is 
capable of being used for video decoding operations. 
Accordingly, existing personal computers usually use the 
25 second or third of the known video decoding techniques 
discussed above. 

Unfortunately, software decoder implementations 
tend to be far slower than hardware implementations 
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operating at comparable clock speeds. This is because 
the CPU circuitry is not optimized for video decoding as 
is the case with a dedicated hardware circuit. In 
addition, the relatively large amount of data which must 
5 be transferred from memory when performing motion 
compensated prediction operations tends to create a 
bottleneck with regard to decoding speed. 

While the use of a graphics processor and a 
10 dedicated video memory can help improve video decoding 

performance, the use of a general purpose CPU to perform 
non-memory intensive decoder operations still normally 
results in a slower decoder implementation than would be 
achieved using dedicated decoder hardware operating at 
15 comparable clock speeds. 



Given the processing capability of CPU's 
currently used in common personal computers, and the 
increasing tendency to run multiple applications at the 
2 0 same time, real time software based video decoding of 
HDTV and other high resolution video images is often 
impractical on modern personal computers. 

In view of the above, it becomes apparent that 
2 5 there is a need for improved methods and apparatus for 
decoding video images in systems, e.g., personal 
computers, which include programmable CPUs. It is 
desirable that at least some of the new methods and/or 
apparatus be capable of being implemented at a lower cost 
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than providing a complete dedicated video decoder circuit 
capable of decoding intra-coded and inter-coded image 
data. It is also desirable that at some of the new 
methods and apparatus be capable of being used with 
5 systems which include a graphics processor for performing 
and/or overseeing memory intensive decoding functions 
such as motion compensated prediction operations. 

SUMMARY OF THE INVENTION 

10 

The present invention is directed to methods 
and apparatus for decoding images. In accordance with 
the present invention video decoder circuitry is combined 
with a programmable processor, e.g., a general purpose 
15 CPU, to perform video decoding operations. A graphics 
processor may be used in combination with the CPU and 
video decoder circuitry of the present invention to 
perform and/ or oversee memory intensive video decoder 
operations . 

20 

The video decoder circuitry of the present 
invention performs one or more non-memory intensive video 
decoding operations such as syntax parsing, variable 
length decoding, DC DCT coefficient reconstruction, 
2 5 motion vector reconstruction, inverse scan operations, 
inverse quantization and/or mismatch control. Memory 
intensive video decoding operations such as performing 
prediction operations using motion vectors and reference 
frames are performed by the programmable processor and/or 



graphics processor operating in conjunction with the 
video decoding hardware of the present invention. 
Because the functions performed by the video decoder 
circuitry of the present invention are normally performed 
at the beginning portions of the inter-coded decoding 
process, e.g., prior to performing motion compensated 
predictions, the video decoder circuitry of the present 
invention may be characterized as a video decoder front 
end processor. In various embodiments, the video decoder 
front end processor includes a complete intra-coded 
decoder. In addition it may, and in various embodiments 
does, include a motion vector reconstruction circuit. 

The decoder circuitry of the present invention 
does not perform memory intensive motion compensated 
predictions. Accordingly, it can be implemented far more 
cheaply than video decoder circuits which perform such 
operations . 

By combining the relatively low cost video 
decoder circuitry of the present invention with a general 
purpose processor, superior decoded video quality can be 
achieved and/ or more images decoded in a given amount of 
time than when a similar general purpose processor is 
used to decode images without the benefit of the decoder 
circuitry of the present invention. Thus, the methods 
and apparatus of the present invention provide a way of 
enhancing a computer system's ability to decode video 
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images at a fraction of the cost of adding a hardware 
decoder capable of fully decoding inter-coded images. 

The video decoder circuitry of the present 
5 invention can be implemented on a separate chip from the 
computer system's CPU. Such a chip can be incorporated 
into a computer card which can be inserted into a 
computer to enhance the system's ability to perform video 
decoding. Alternatively, the video decoder circuitry of 
10 the present invention can be incorporated directly into a 
processor thereby providing the video decoder circuitry 
of the present invention and a programmable processor on 
a single semiconductor chip. 

15 Various additional features and advantages of 

the present invention will be apparent from the detailed 
description which follows. 

2 0 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 illustrates a known MPEG-2 video 

decoder. 

25 Figs. 2 and 3 illustrate exemplary computer 

systems of the present invention wherein a video front 
end processor is used in combination with a CPU and 
software to decode encoded video . 
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Fig. 4 illustrates the front end processor of 
the present invention, used in the Fig, 2 and 3 
embodiments, in greater detail . 

5 

DETAILED DESCRIPTION 

As discussed above, the present invention is 
directed to methods and apparatus for decoding images, 

10 e.g., MPEG-2 video images. The methods and apparatus of 
the present invention are well suited for use with 
systems which include a programmable processor, e.g., a 
computer system which includes a CPU. They are also well 
suited for use in systems which include, in addition to a 

15 CPU, a graphics processor for performing or overseeing 
memory intensive video decoding operations. 

In contrast to prior art techniques which used 
a general purpose processor to perform non-memory 

2 0 intensive video decoding operations, the inventors of the 
present application realized that there can be processing 
speed and quality advantages to performing non-memory 
intensive video decoding operations in hardware. It was 
further recognized that a hardware circuit to perform 

25 such decoder operations could be implemented at 

relatively little cost and could be used in conjunction 
with a general purpose processor, or incorporated into a 
general purpose processor, to facilitate video decoding 
operations . 
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By using dedicated video decoder hardware 
circuitry to perform some, most, or all of the non-memory 
intensive operations associated with decoding one or more 
5 images, the ability of a computer system to decode still 
images and video is enhanced at a fraction of the price 
of providing a full hardware video decoder circuit 
capable of decoding inter-coded and intra-coded images. 

10 Figure 2 illustrates a computer system 300 

implemented in accordance with one embodiment of the 
present invention. As illustrated the system 300 
includes a video decoder front end processor circuit 3 02 
used for performing non-memory intensive video decoding 

15 operations in accordance with the present invention, a 
programmable processor (CPU) 316, a display device 210, 
display adapter 312, keyboard 234, a first memory bus 
229, a second memory bus 229', and a system bus 228. 

20 The system 300 further includes a secondary 

(L2) cache 236 which is coupled to the CPU 316 via the 
first memory bus 229 and to the main memory 326 via the 
second memory bus 229 ' . The system bus 228 couples a 
network adapter 224 and a peripheral device 322 to one 

25 another and to the CPU 316. The network adapter 224 is 
used to couple the system 300 to a digital network via 
which encoded video data may be received. 
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The peripheral device 322 is coupled to the 
video front end processor circuit 302 in addition to 
being coupled to the system bus 228. In this manner, the 
peripheral device 322 which may be, e.g., a video disc 
player, can supply encoded video directly to the video 
front end processor circuit 302. While a single 
peripheral device 322 is shown in the system 3 00 it is to 
be understood that the system can, and in various 
embodiments does, include multiple peripheral devices 
322. 

In the system 300, the CPU 316 includes a cache 
320 and an I/O interface 318. The I/O interface 318 
couples the CPU 316 to the video front end processor 
circuit 302, display adapter 312, keyboard 234, system 
bus 228 and memory bus 229. The I/O interface 318 serves 
to electrically connect the devices coupled thereto and 
to convert between internal signaling formats used by the 
CPU 316 and external devices coupled thereto. The CPU 
316 performs processing operations under control of 
routines stored in the main memory 326. The stored 
routines include a video decoder routine 330 and various 
other routines, e.g., spread sheet, word processor, web 
browser, etc. As will be discussed below, unlike the 
prior art video decoder routines which were responsible 
for controlling a CPU to fully decode inter-coded video 
data, the video decoder routine 330 of the present 
invention is designed to control the CPU so that it 
interacts with the video decoder front end processor 
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circuit 3 02 in such a manner as to produce fully decoded 
video data. Thus, under control of the routine 330 the 
CPU 316 performs video decoding operations not performed 
by the video processor circuit 302, e.g., motion 
compensated prediction operations. 

In the Fig. 2 embodiment, a display adapter 312 
is used to couple the CPU 316 to the display device 210. 
The display adapter 312 converts video signals output 
from the CPU 316 into a format used by the display device 
210. The signals, e.g., video images, output by the 
display adapter are displayed on the display device 210. 

Operation of the video processor circuit 3 02 
will be described in detail below with regard to Fig. 4. 
However, generally speaking, the video processor circuit 
3 02 receives configuration and other set-up information 
from the CPU 316 and processes encoded video received 
from the peripheral device 322 or CPU. The performed 
processing includes, e.g., syntax parsing, variable 
length decoding, and/ or other non-memory intensive video 
decoding operations. The video data, e.g., decoded 
intra-coded image data and/or partially decoded inter- 
coded image data, generated by video processor circuit 
3 02, is passed to the CPU 316. In the Fig. 2 embodiment, 
the CPU 316 is responsible for completing the image 
decoding process to the extent that additional decoding 
operations are required. For example, in the case of 
inter-coded image data, the CPU 316 under control of the 
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video decoder routine 330 performs motion compensated 
predictions using reconstructed motion vector information 
received from the video processor circuit 302. As will 
be discussed below, in various embodiments, the video 
5 front end processor circuit 302 is capable of fully 
decoding intra-coded image data. Accordingly, in the 
case of intra-coded frames, the CPU 316 need not decode 
the image data output by the video front end processor 
circuit 302. However, when processing video data 

10 including both intra-coded and inter-coded frames, the 
previously decoded frame data output by the processor 
circuit 302 is normally stored on a temporary basis. The 
temporary storage of decoded image data allows for frame 
re-sequencing and renders the image data available in the 

15 event that it is needed as reference frame data in 

performing a motion compensated prediction operation. 

Fig. 3 illustrates another exemplary embodiment 
of the present invention. The circuitry 400 illustrated 

2 0 in the Fig. 3 embodiment is similar to that of the Fig. 2 
embodiment. However the display adapter 312 is replaced 
with a graphics processor 412 capable of performing 
and/or overseeing memory intensive video decoder 
operations. In addition, in the Fig. 3 embodiment a 

25 graphics memory 214 and a video bus 232 are included to 
facilitate performing memory intensive video operations 
such as motion compensated prediction. The graphics 
memory 214 is used for storing reference frame data used 
for motion compensated predictions while the video bus 
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232 is used for exchanging information between the main 
memory and video memory at a high rate of speed and 
without interfering with communications over the system 
bus 22 8. 

Fig. 4 illustrates a video decoder front end 
processor circuit 302 which may be used in the systems 
illustrated in Figs. 2 and 3. As illustrated, the 
processor circuit 302 includes an input buffer 502, a VLD 
circuit 504, memory 520, an inverse scan circuit 506, 
inverse quantization circuit 508, inverse DCT circuit 
510, an optional downsampler 512, a motion vector 
reconstruction circuit 518, a MUX 514, an output buffer 
516, and an output control circuit 522. 

While the circuit 302 includes memory 52 0, it 
is a relatively small amount of memory compared to the 
graphics memory 214 or main memory 326. This is because 
the memory 520 is used for storing control, configuration 
and, context information relevant to decoding operations 
as opposed to the actual frames. One kilobyte is an 
exemplary size of the memory 520 in the case of 8 bits 
per byte. The memory 520 includes a set of context, 
control, and configuration information 525, 527 for each 
of N bitstreams which are processed by the circuit 302. 
In addition to, or as part of, the memory 520, video 
buffers, e.g., processing buffers, may be included on the 
video decoder front end processor circuit 3 02. 
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The input buffer 502 is used to temporarily 
store video data and other input received by the video 
circuit 302 until the VLD circuit 504 is ready to process 
the input. The input can include encoded video data, 
5 configuration, and/or video bitstream information 

received from the peripheral device 322 and/or CPU 316. 
The encoded video data may be, e.g., an MPEG-2 packetized 
elementary stream (PES) or a video elementary stream 
which is produced by depacketizing a PES stream. Thus, 

10 the encoded video data input to the video circuit 3 02 may 
include, for example, coded motion vectors, coded 
quantized blocks of DCT coefficients, macroblock type 
information and other ancillary data such as coded block 
pattern information, coded macroblock position 

15 information, field select signals, and motion type 

information. Received configuration information may 
include, e.g., a reset signal, down sampling mode control 
information in the case of a system which supports 
downsampling, data output format (e.g., 8 or 9 bit output 

2 0 format information) , and/or image region information 
indicating for example, whether a border region or a 
center region of a letterbox image is being sent to the 
circuit 302 for processing. 

25 The VLD circuit 504 receives the input to the 

circuit 302 from the input buffer 502, The VLD circuit 
504 is responsible for performing a variable length 
decoding operation on received variable length encoded 
data. The VLD circuit 504 also identifies ancillary 
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data. In addition, it identifies configuration, 
control, and video context information. Ancillary data 
and context information are supplied from the VLD circuit 
504 to a third input of MUX 514. The configuration, 
5 control and video context information are also passed to 
the memory 520 and to the output control circuit 522. 

Configuration, context, and/or control 
information can also be passed directly from the CPU 316 
10 to the memory 520 thereby avoiding the need to transmit 

the information with the encoded video data being sent to 
the VLD circuit 504. 

The memory 520 stores and outputs context 
15 information for each bitstream being decoded. The 

context information stored for each bitstream includes, 
e.g., high level syntax picture level and sequence level 
information. Exemplary stored picture level information 
includes frame type information (I, P, B frame 
20 information) while exemplary stored sequence level 

information includes vertical size, horizontal size, and 
frame rate. The video circuit 302 can be used to support 
decoding of multiple bitstreams, e.g., in order to 
support picture-in-picture capabilities. By storing 
25 context, control and configuration information 525, 527 
for each of the multiple bitstreams being decoded the 
data need not be transmitted multiple times to the 
circuit 302. Each time a different bitstream is to be 
decoded, for which information is already stored in the 
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memory 52 0, stored information associated with the 
bitstream to be decoded is supplied to the VLD circuit 
504, motion vector reconstruction circuits 518 and 
downsampling circuit 512. In this manner, the video 
5 circuit 3 02 can be reconfigured to processes a different 
bitstream in a fast and efficient manner. For example, 
in response to the stored control information when 
processing image data intended to be used as a main 
picture, the downsampling circuit 512 may be disabled. 

10 However, in response to stored control information when 
processing inset picture data the downsampling circuit 
512 may be enabled. Thus, data reduction operation may 
be performed on inset picture data but not main picture 
data in response to control information received from the 

15 memory 52 0. 

The memory 52 0 also supplies the control 
information associated with a bitstream being processed 
to the output control circuit 522. The output control 

20 circuit 522 is coupled to a control input of the MUX 514 
and a control input of the output buffer 516. The output 
control circuit 522 controls which data, e.g., decoded 
image data, motion vectors, ancillary data, etc. are 
output. The circuit 522 also controls when the 

25 information is output from the video circuit 302. The 
output control circuit 522 does this by controlling the 
MUX 514 and output buffer 516. 
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The VLD circuit 504 outputs image data 
generated by performing a variable length decoding 
operation. The image data output by the VLD circuit 5 04 
is supplied to the inverse scan circuit 506 which 
5 performs the inverse of the scan conversion process 
performed at encoding time. Next, the video data is 
processed by the inverse quantization circuit 508 and 
then the inverse DCT circuit 510. The inverse 
quantization circuit performs an inverse quantization 
10 operation on received data while the inverse DCT circuit 
performs an inverse discrete cosine transform operation 
on received data. The decoded image data output by the 
IDCT circuit 510 is supplied to the downsampling circuit 
512 . Downsampling is performed in cases where the 
15 received control and/or configuration information 
indicates that it should be applied. For example 
downsampling may be applied to inset picture data and/or 
HDTV images when the images will be displayed at lower 
than their full resolution. The resulting decoded image 
2 0 data, corresponding to intra-coded frames prediction 
residuals, is supplied from the downsampler 512 to a 
first input of the MUX 514. 

The VLD circuit 504 outputs motion vector 
25 information including, e.g., motion vector values encoded 
using differential encoding, to the motion vector 
reconstruction circuit 518. The reconstruction circuit 
518 reconstructs motion vectors from the received 
information, e.g., by performing a reverse differential 
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coding operation. The reconstructed motion vectors are 
supplied to a second input of the MUX 514. 



As discussed above, the output control circuit 
5 522 control the MUX 514. In response to the control 

signal from the control circuit 522, the MUX 514 outputs 
the data from one of its inputs. This information is 
stored in the output buffer 516 prior to being 
transmitted to the CPU 316 at a rate controlled by the 

10 output control circuit 522. Thus, video processor 

circuit 3 02 supplies the CPU 316 with decoded image data, 
e.g., pixel values, corresponding to intra-coded frames 
and intra-coded image portions. With regard to inter- 
coded image portions it supplies, e.g., reconstructed 

15 motion vectors, coded block pattern information or 

pattern code information, macroblock position information 
and/or pixel residual values used in performing motion 
compensated predictions. In cases where the video 
processor circuit 302 does not perform an IDCT operation, 

20 it may also output to the CPU 316 DCT coefficients. 

The CPU 316 or CPU 316 operating in conjunction 
with a graphics processor 412 completes the decoding of 
inter-coded images using the video data supplied thereto 
25 from the video decoder front end processor circuit 3 02 of 
the present invention. 

In the above described manner, the video front 
end processor circuit 3 02 of the present invention can be 
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used to enhance and/ or increase the speed at which a 
system including a programmable processor can perform 
video decoding operations* 

5 While the video decoder front end processor 

circuit 302 of the present invention has been described 
as performing several operations, it is to be understood 
that one or only a few of the described operations may be 
performed by a video front end processor with the 
10 remaining operations being performed by a programmable 

CPU or a programmable CPU operating in combination with a 
graphics processor. The decoder front end processor may 
reside on a graphics processor board or on a separate 
printed circuit board, e.g., a computer plug in board. 

15 

Numerous additional embodiments, within the 
scope of the present invention, will be apparent to those 
of ordinary skill in the art in view of the above 
description and the claims which follow. 
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WHAT IS CLAIMED IS: 

1 1. A method of decoding encoded image data comprising 

2 the steps of: 

3 operating a decoder circuit implemented in 

4 hardware to perform at least one non-memory intensive 

5 image decoding operation to generate, from the encoded 

6 image data, a first set of processed image data, the at 

7 least one non-memory intensive image decoding operation 

8 being an operation in the group of operations consisting 

9 of a variable length decoding operation, an inverse scan 

10 conversion operation, and an inverse quantization 

11 operation; 

12 supplying the first set of processed image data 

13 generated by the decoder circuit to a programmable 

14 processor; and 

15 operating the programmable processor to perform 

16 at least one additional image decoding operation using 

17 the first set of processed image data. 
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1 2. The method of claim 1, wherein the step of operating 

2 the decoder circuit, includes the step of performing at 

3 least two additional operations from the group of 

4 operations consisting of a variable length decoding 

5 operation, an inverse scan conversion operation, an 

6 inverse quantization operation, an inverse discrete 

7 cosine transform operation, and a data reduction 

8 operation, the two additional operations being different 

9 from said at least one non-memory intensive operation. 

1 3. The method of claim 1, wherein the step of operating 

2 the decoder circuit further includes: 

3 operating the decoder circuit to perform a data reduction 

4 operation. 

1 4. The method of claim 2, wherein the step of operating 

2 the decoder circuit further includes: 

3 operating the decoder circuit to perform a data 

4 reduction operation. 
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1 5. The method of claim 2, wherein the step of operating 

2 the programmable processor to perform at least one 

3 additional image decoding operation includes the step of: 

4 operating the programmable processor to perform a 

5 motion compensated prediction operation. 

1 6, The method of claim 5, wherein the step of operating 

2 the programmable processor to perform at least one 

3 additional image decoding operation further includes the 

4 step of: 

5 operating the programmable processor to combine 

6 decoded image data produced by performing the motion 

7 compensated prediction operation with decoded residual 

8 image data to produce a set of decoded image data 

9 representing reconstructed pixels. 

1 7. The method of claim 1, wherein the step of operating 

2 the programmable processor to perform at least one 

3 additional image decoding operation includes the step of: 

4 operating the programmable processor to combine 

5 decoded image data produced by performing a motion 

6 compensated prediction operation with decoded intra-coded 

7 image data to produce a set of decoded image data 

8 representing a complete frame. 

1 8. The method of claim 2, wherein the programmable 

2 processor is coupled to a graphics processor, the method 

3 further comprising the step of: 



operating the graphics processor to perform a motion 
compensated prediction operation using data included in 
the first set of processed data. 

9. The method of claim 8, wherein the step of operating 
the programmable processor to perform at least one 
additional image decoding operation further includes the 
step of: 

operating the programmable processor to combine 
decoded image data produced by performing the motion 
compensated prediction operation with decoded residual 
image data to produce a set of decoded image data 
representing reconstructed pixels. 

10. The method of claim 8, further comprising the step 
of: 

storing in the decoder circuit multiple sets of 
context information, each set of stored context 
information corresponding to a different one of a 
plurality of encoded data streams processed by the 
decoder circuit. 

11. The method of claim 1, further comprising the step 
of: 

storing in the decoder circuit multiple sets of 
context information, each set of stored context 
information corresponding to a different one of a 
plurality of encoded data streams processed by the 
decoder circuit. 
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1 12. The method of claim 11, further comprising the step 

2 of: 

3 operating the decoder circuit to access the 

4 stored set of context information corresponding to an 

5 encoded data stream when the data stream is to be 

6 processed by the decoder circuit. 

1 13. /The method of claim 12, wherein each set of stored 

2 context information includes encoded data stream syntax 

3 information. 

1 14. A method of decoding encoded image data including 

2 inter- coded image data and intra- coded image data, the 

3 method comprising the steps of: 

4 operating an intra-coded video decoder circuit 

5 implemented in hardware to decode said intra-coded image 

6 data and to output prediction residual data produced from 

7 said encoded image data; and 

8 controlling a programmable processor to perform 

9 an inter-coded decoding operation using said prediction 
10 residual image data. 

1 15. The decoding method of claim 14, wherein the step of 

2 controlling a programmable processor to perform a 

3 decoding operation includes the step of: 

4 controlling the programmable processor to 

5 perform a motion compensated prediction operation. 
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1 16. The decoding method of claim 14, wherein the step of 

2 controlling a programmable processor to perform an inter- 

3 coded decoding operation includes the step of: 

4 operating the programmable processor to control 

5 the supply of motion vector information to a graphics 

6 processor . 

1 17. The decoding method of claim 14, further comprising 

2 the step of: 

3 controlling a graphics processor coupled to said 

4 programmable processor to perform a motion compensated 

5 prediction operation. 

1 18. The decoding method of claim 14, wherein the step of 

2 operating the decoder circuit to decode said intra-coded 

3 image data includes performing a complete decoding 

4 operation on said intra-coded image data to produce fully 

5 decoded image data therefrom. 

1 19. A method of decoding encoded image data comprising 

2 the steps of: 

3 operating a decoder circuit implemented in 

4 hardware to perform non-memory intensive image decoding 

5 operations to generate, from the encoded image data, a 

6 first set of processed image data, at least one of said 

7 non-memory intensive image decoding operation being a 

8 data reduction operation; 
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9 supplying the first set of processed image data 

10 generated by the decoder circuit to a programmable 

11 processor; and 

12 operating the programmable processor to perform 

13 at least one additional image decoding operation using 

14 the first set of processed image data. 

1 20. A system for decoding encoded image data including 

2 intra-coded image data and inter-coded image data, the 

3 system comprising: 

4 an intra-coded data decoding circuit for decoding 

5 intra-coded image data; 

6 a programmable processor coupled to the intra-coded 

7 data decoding circuit; and 

8 a memory including a video decoding routine for 

9 controlling the programmable processor to perform at 

10 least one inter-coded data decoding operation. 

1 21. The system of claim 20, further comprising: 

2 a graphics processor coupled to the 

3 programmable processor. 

1 22. The system of claim 20, wherein the intra-coded data 

2 decoding circuit includes: 

3 an inverse discrete cosine transform circuit and an 

4 inverse quantization circuit. 



1 23. The system of claim 22, further comprising: 
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2 a motion vector reconstruction circuit for 

3 reconstructing motion vectors included in said inter- 

4 coded image data, the motion vector reconstruction 

5 circuit being coupled to said programmable processor. 

1 24. The system of claim 23, wherein said intra-coded 

2 data decoding circuit and motion vector reconstruction 

3 circuit are implemented on a first semiconductor chip and 

4 wherein said programmable processor is implemented on a 

5 second semiconductor chip. 

1 25. The system of claim 23, wherein said intra-coded 

2 data decoding circuit, motion vector reconstruction 

3 circuit and programmable processor are implemented on a 

4 single semi-conductor chip. 

1 26. The system of claim 20, wherein the intra-coded data 

2 decoding circuit includes a variable length decoding 

3 circuit for processing both intra-coded and inter-coded 

4 image data. 

1 27. The system of claim 26, 

2 wherein the intra-coded data decoding circuit 

3 further includes an inverse discrete cosine transform 

4 circuit and an inverse quantization circuit; 

5 wherein the system further includes a motion 

6 vector reconstruction circuit for reconstructing motion 

7 vectors included in said inter-coded image data coupled 

8 to said programmable processor; and 



-32- 

9 wherein the variable length decoding circuit includes 

10 means for output ting intra-coded data to the inverse 

11 quantization circuit and means for outputting motion 

12 vector information to the motion vector reconstruction 

13 circuit. 

1 28. The system of claim 20, wherein the intra-coded data 

2 decoding circuit and the programmable processor are 

3 implemented on two separate semi-conductor chips. 

1 29. An apparatus for processing encoded image data 

2 including motion vector information, the apparatus 

3 comprising: 

4 a motion vector reconstruction circuit for 

5 performing motion vector reconstruction operations using 

6 motion vector information included in the encoded image 

7 data; and 

8 means for outputting to a programmable 

9 processor reconstructed motion vectors generated by the 

10 motion vector reconstruction circuit. 

1 30. The apparatus of claim 29, further comprising: 

2 said programmable processor coupled to the 

3 means for outputting; and 

4 a memory device coupled to said programmable 

5 processor, the memory device including a video decoding 

6 routine used to control said programmable processor to 

7 perform a video decoding operation using reconstructed 

8 motion vectors received from the means for outputting. 



ABSTRACT 



Video decoder methods and apparatus are 
described. In accordance with the invention, hardware 
decoder circuitry, e.g., intra-coded image decoding 
circuitry and motion vector reconstruction circuitry, is 
used in combination with a general purpose processor, 
e.g., Pentium processor, to perform video decoding 
operations. The video decoder hardware circuitry of the 
present invention is responsible for performing non- 
memory intensive functions. The general purpose 
processor or a general purpose processor operating in 
conjunction with a graphics processor are used to perform 
memory intensive video decoding operations such as motion 
compensated predictions. The video decoding hardware 
circuitry of the present invention can be implemented as 
a separate physical device, e.g., chip, or can be 
implemented on the same physical chip as a general 
purpose processor with which it works. By using the 
video decoding hardware circuitry of the present 
invention in combination with a CPU, a computer system's 
ability to perform video decoding operations can be 
significantly increased at little cost in terms of 
additional hardware. 
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